Computer-assisted evaluation of CSCL chat conversations

Although instant messaging is a very popular tool for collaboration and it has been used for a wide variety of CSCL tasks, there are only a few applications for assisting the tutors in evaluating the conversations of the students. Due to the difficulty of this task, chat is seldom used in a formal education context. In order to tackle this problem, several applications were developed which assist the tutors when evaluating chat conversations. This paper presents a comparison of the evaluation results when using these applications on a set of three multi-user chat sessions. Introduction Instant messaging has been proven to be an effective way of undertaking Computer Supported Collaborative Learning (Stahl, 2006) because it is a simple and efficient synchronous communication tool which facilitates the construction of complex ideas as a result of group discourse. It is also suited for larger groups because, unlike verbal conversations – either face-to-face or videoconferences, it permits the development of different discussion threads at a particular moment in time. Therefore, many environments and tools which have been developed for CSCL during the last 15 years have included instant messaging facilities (Baker & Lund, 1996). Various enhancements to standard instant messenger technology have been proposed and CSCL chats tools have introduced facilities such as explicit referencing of previous utterances or concepts (Mühlpfordt & Wessner, 2005), whiteboards and concept map drawing board. Nevertheless, with a few exceptions, the usage of chat was not adopted in a formal educational context, mostly due to the difficulty of assessing the participants’ participation to conversations. In order to solve this issue, there is a need for tools that analyze chat discussions and provide feedback to the tutor. This paper presents a comparison regarding the evaluation of chat conversations by two groups of tutors: those analyzing the chats using specialized tools and those who are not using any analysis software. The next section describes the chat experiment and the main features of the tools used for analysis. Then, we shall present the results of evaluating the participants in the chat by four different tutors, as well as the automatic grading offered by the software. The paper ends with conclusions and references. The Chat Experiment and the Analysis Tools Several chat experiments were undertaken in order to assess the students’ knowledge and understanding of certain topics and concepts covered during the course. In one of these experiments, undergraduate students participating in the Human-Computer Interaction course were grouped in teams of 3-4 members and asked to chat about collaborative web technologies. The conversation was composed of two distinct stages: during the first stage, there was a debate to decide which web technology among instant messaging, discussion forum, blog and wiki is the most powerful and well suited, while the second one consisted of a cooperating effort in order to find the best way to integrate all of these technologies under the same platform. Thus, the participants were engaged in both a competitive and a cooperative discourse in the same chat, without any moderator. The students used the ConcertChat (1) system which offers an explicit referencing mechanism that is very useful in chats with more than two participants, as it provides linking to previous utterances or phrases. Two distinct tools were used to analyze the chat logs in order to facilitate the evaluation of the participants: Polyphony Analyser (Trausan et al., 2007) and ASAP (Dascalu et al., 2008). Both tools combine natural language processing and social network analysis (SNA) techniques in order to provide feedback about a chat conversation, While the Polyphony Analyser mostly uses NLP techniques combined with the lexical ontology WordNet (2), ASAP is oriented more towards SNA. The former combines ideas specific to the sociocultural paradigm (Vygotsky, 1978) and to the dialogistic ideas of Bakhtin (1973), as well as to the classical cognitive paradigm that uses ontologies and knowledge-based processing. This application can be used to facilitate the discovery of important semantic and social data from the chat: the extent to which the topics were covered by the participants, the main subjects of the discussion, an evaluation of the competence of each participant, a graphical view of the chat that can be useful in evaluating the implication of each participant and the degree of debate. Moreover, the tools can be used to discover new implicit references between utterances and it offers a better visualization of the discussion threads. ASAP (An Advanced System for Assessing Chat Participants) uses the social network of the participants and the utterances graph which result from the succession of turns in the chat and the explicit references between them. Thus, the social network can be modeled as a graph taking the participants as vertices and the references between utterances as edges. In order to evaluate the competency of each participant, several qualitative and quantitative factors are being considered such as the characteristics of the social network (e.g. closeness, rank), the speech acts that are present in each utterance and the importance of the words in each utterance. The evaluation of a collaborative chat conversation and the provision feedback (including grading) to the students is a difficult and time-consuming task for the tutor, especially because information regarding both the collaboration process and the content has to be taken into account. Moreover, when distinct tutors evaluate a certain number of chats, it is mandatory to establish a common set of rules, in order to ensure a homogenous grading process. To this extent, it is important to notice that qualitative and prompt feedback is crucial for the students in order to be aware of their mistakes and try to correct them. The following section will present a comparative study that shows that using the tools described greatly reduces the time needed to evaluate a chat conversation and improves the feedback generated. Comparison of the Evaluation Results Three collaborative chat conversations, involving four students each, from the experiment described in the previous section have been chosen to be analyzed by four distinct evaluators who are tutors for the HCI course. The three tables below offer the grading of the participants in each chat offered by the evaluators and by the two analysis tools. Furthermore, evaluator 3 and 4 used the feedback provided by these applications, while evaluators 1 and 2 did not. Chat conversations 4 and 36 are considered positive examples both regarding the content and the degree of collaboration, while chat 34 is a negative one. Table 1: Evaluation results for chat conversation 4. Chat No.4 Student 1 Student 2 Student 3 Student 4 Evaluator 1 9 8 7 8 Evaluator 2 10 9 7 6.5 Evaluator 3 8 8.5 8 9 Evaluator 4 9.5 10 6.5 8 Evaluator average 9.125 8.875 7.125 7.875 Polyphony 10 8.23 6.50 8.17